perf(gb300): Refresh Qwen3.5 FP4 GB300 dynamo-trt STP/MTP recipes from srt-slurm / 刷新基于 srt-slurm 的 Qwen3.5 FP4 GB300 dynamo-trt STP/MTP 配方 - #2730
Conversation
…m srt-slurm / 刷新基于 srt-slurm 的 Qwen3.5 FP4 GB300 dynamo-trt STP/MTP 配方
Replace the qwen3.5-fp4-gb300-dynamo-trt and qwen3.5-fp4-gb300-dynamo-trt-mtp
search spaces with a new locally validated 10-point STP and 10-point MTP
sweep sourced from srt-slurm (recipes/trtllm/qwen3.5-fp4/inferencex/gb300/{mtp,stp}).
Bumps the TensorRT-LLM image to 1.3.0rc24, switches STP kv-p2p-transfer to
nixl to match the new recipes, and moves the checkpoint to
nvidia/Qwen3.5-397B-A17B-NVFP4-V2.
用来自 srt-slurm 仓库(recipes/trtllm/qwen3.5-fp4/inferencex/gb300/{mtp,stp})
的新配方,替换 qwen3.5-fp4-gb300-dynamo-trt 与 qwen3.5-fp4-gb300-dynamo-trt-mtp
的搜索空间,分别包含 10 个已本地验证的 STP 与 MTP 配置点。将 TensorRT-LLM
镜像升级至 1.3.0rc24,将 STP 的 kv-p2p-transfer 切换为 nixl 以匹配新配方,
并将模型切换到 nvidia/Qwen3.5-397B-A17B-NVFP4-V2。
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| - "STP: 10 configs from 1P2D (conc 42) to 24P1D (conc 8192)." | ||
| - "MTP: 10 configs from 1P2D (conc 20) to 26P1D (conc 4301, 2-token draft); the 24P1D/conc 8192 tail point shares its recipe with the STP curve, where MTP no longer helps at saturation." | ||
| - "Recipes sourced from srt-slurm (recipes/trtllm/qwen3.5-fp4/inferencex/gb300/{mtp,stp})." | ||
| pr-link: TBD |
There was a problem hiding this comment.
🔴 New entry uses 'pr-link: TBD' but validate_perf_changelog.py only accepts the canonical PR URL or the literal placeholders 'XXX' / '.../pull/XXX' (PR_LINK_PLACEHOLDERS in utils/validate_perf_changelog.py:24-27).
Extended reasoning...
The perf-changelog CI gate calls validate_added_pr_link on the appended entry; since 'TBD' is neither the canonical https://github.com/SemiAnalysisAI/InferenceX/pull/ link nor in PR_LINK_PLACEHOLDERS, validate_perf_changelog.py raises ChangelogValidationError and the changelog-gate check fails on this very PR, blocking merge until the author replaces TBD with 'XXX' or the real PR link.
Verification: normal — the appended perf-changelog entry (perf-changelog.yaml diff tail) ends with pr-link: TBD, which no validation path accepts. In utils/validate_perf_changelog.py, the only accepted values are CANONICAL_PR_LINK (lines 21-23, .../pull/\d+) and PR_LINK_PLACEHOLDERS (lines 24-27, {"XXX", ".../pull/XXX"}). validate_added_pr_link (lines 144-160) raises ChangelogValidationError for "TB
| model: nvidia/Qwen3.5-397B-A17B-NVFP4-V2 | ||
| model-prefix: qwen3.5 | ||
| runner: gb300 | ||
| precision: fp4 |
There was a problem hiding this comment.
🟡 qwen3.5-fp4-gb300-dynamo-trt sets scenario-level kv-p2p-transfer: nixl, but 4 of the 10 new STP recipe files it references (ctx1_gen2_tep8_batch16_eplb0_mtp0_42.yaml, ctx1_gen4_tep8_batch1_eplb0_mtp0_8.yaml, ctx1_gen4_tep8_batch4_eplb0_mtp0_24.yaml, ctx4_gen3_tep8_batch64_eplb0_mtp0_210.yaml) still set cache_transceiver_config.backend: UCX internally; same pattern in qwen3.5-fp4-gb300-dynamo-trt-mtp (kv-p2p-transfer: nixl at line ~6560) where 5 of 10 mtp recipes (e.g. ctx1_gen2_tep8_batch8_eplb0_mtp3_20.yaml, ctx7_gen1_dep16_batch32_eplb0_mtp3_666.yaml) also use UCX.
Extended reasoning...
kv-p2p-transfer is a free-text metadata field (utils/matrix_logic/validation.py) surfaced in dashboards/reports as the KV transfer backend for the run, but it is never cross-checked against the CONFIG_FILE's actual cache_transceiver_config.backend. For these mismatched entries, anyone reading benchmark results or the perf-changelog will believe NIXL was used when the run actually executed with UCX, contradicting the PR's own stated goal ('switch STP's kv-p2p-transfer... to match the new recipes' NIXL cache-transceiver backend') and misleading performance comparisons/regressions attributed to NIXL vs UCX.
Verification: Severity: nit (metadata/reporting inaccuracy; nothing functionally breaks). The mismatch is real and reachable. configs/nvidia-master.yaml sets scenario-level kv-p2p-transfer: nixl for both qwen3.5-fp4-gb300-dynamo-trt (line ~6399) and qwen3.5-fp4-gb300-dynamo-trt-mtp (line ~6560). But 4 of the 10 referenced STP recipes and 5 of the 10 MTP recipes set cache_transceiver_config.backend: UCX internal
# Conflicts: # perf-changelog.yaml
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32901704325 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32901704325 |
|
/reuse-sweep-run 32901704325 |
|
As a PR reviewer and CODEOWNER, I have reviewed this and have:
Additional detail section:
Signed: |
✅✅✅ Verdict: PASS ✅✅✅✅ Check 0 (CODEOWNER): PASS — |
|
/stage-results 32901704325 |
|
@cquil11 staged run 32901704325: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-08-25~r32901704325 This run remains available across future |
|
/reuse-sweep-run |
中文：同步 main，并恢复仅追加且字节敏感的性能变更记录。
4 participants
Summary / 摘要
qwen3.5-fp4-gb300-dynamo-trt(STP) andqwen3.5-fp4-gb300-dynamo-trt-mtp(MTP) search spaces with a new locally validated 10-point STP + 10-point MTP sweep sourced from srt-slurm (recipes/trtllm/qwen3.5-fp4/inferencex/gb300/{mtp,stp}).recipes/trtllm/qwen3.5-fp4/inferencex/gb300/{mtp,stp})的新配方,替换qwen3.5-fp4-gb300-dynamo-trt(STP)与qwen3.5-fp4-gb300-dynamo-trt-mtp(MTP)的搜索空间,共 10 个 STP 配置点与 10 个 MTP 配置点。1.3.0rc18→1.3.0rc24, switch STP'skv-p2p-transferfromucxtonixlto match the new recipes' NIXL cache-transceiver backend, and switch the checkpoint tonvidia/Qwen3.5-397B-A17B-NVFP4-V2.1.3.0rc18→1.3.0rc24,STP 的kv-p2p-transfer由ucx切换为nixl以匹配新配方的 NIXL cache-transceiver 后端,并将模型切换为nvidia/Qwen3.5-397B-A17B-NVFP4-V2。perf-changelog.yamlentry for both config keys.perf-changelog.yaml条目。Test plan / 测试计划
python3 -c "import yaml; yaml.safe_load(open('configs/nvidia-master.yaml'))"and same forperf-changelog.yaml— both parse cleanly.utils/matrix_logic/generate_sweep_configs.py full-sweep --config-files configs/nvidia-master.yaml --framework dynamo-trt— generates all 20 qwen3.5 scenarios with correct topology andCONFIG_FILEmapping.full-sweep-fail-fastapplied).🤖 Generated with Claude Code
Note
Low Risk
Benchmark and orchestration YAML only; no application logic changes. Risk is operational (wrong recipe → failed Slurm runs or misleading perf numbers).
Overview
Refreshes Qwen3.5-397B FP4 GB300 disaggregated TensorRT-LLM + Dynamo benchmark recipes and matrix entries for 8k/1k
sa-bench, replacing the older rc18 / UCX-heavy sweep with a locally validated set from srt-slurm.Runtime and checkpoint: Matrix configs move to
tensorrt-llm/release:1.3.0rc24, modelnvidia/Qwen3.5-397B-A17B-NVFP4-V2, and STPkv-p2p-transfer: nixl(wasucx).launch_gb300-nv.shpins srt-slurmv1.0.72(wasv1.0.29) and still copies recipes frombenchmarks/multi_node/srt-slurm-recipes/trtllm/qwen3.5/gb300-fp4/disagg.Recipe corpus: Adds/rewrites YAML under
disagg/8k1k/mtp/anddisagg/8k1k/stp/with consistent naming (disagg-gb300-*), Dynamo wheel1.4.0.dev20260807, staggered ctx load, and NIXL or UCX cache transceivers (max_tokens_in_buffer: 16384). Removes many legacyctx*dep2_*rc18 configs tied to old topologies (TP2 prefill, TP16 decode, etc.).Search space (
nvidia-master.yaml):qwen3.5-fp4-gb300-dynamo-trt— 10 STP points (e.g. conc 42–8192, topologies from 1P2D through 24P1D).qwen3.5-fp4-gb300-dynamo-trt-mtp— 10 MTP points (conc 20–4301, mostly MTP draft-len 3; 26P1D uses draft-len 2; 24P1D / 8192 reuses the STP recipe as a non-MTP saturation tail).perf-changelog.yamldocuments both config keys.Reviewed by Cursor Bugbot for commit 6fff90a. Bugbot is set up for automated code reviews on this repo. Configure here.